08/05/2005 15:19 FAX 6176950892 WSGL @008 



Application No. 09/869,638 
Filed: February 8, 2002 
TC Art Unit: 2621 
Confirmation No. : 5891 

REMARKS 

Pending claims 1-3, 6-12 and 14 have been rejected as obvious 
over Luck et al . in view of Diouf et al . [sic, Watkinl and 
Watanabe. Claims 4, 5, 13 and 15 have been rejected as? obvious 
over Luck et al . , in view of Diouf et al - and Watanabe, and 
further in view of Hemstreefc, III et al . These rejections are 
respectfully traversed for the reasons given below and 
reconsideration is requested - 

The present invention is directed to a method for the 
automatic analysis of microscope images of biological objects/ 
which is particularly useful for the analysis of fluorescence 
images of cells. In making the rejection, the Examiner has 
characterized the disclosure of the primary reference, Luck et 
al., in relation to the various steps of Applicants 1 method claim 
1. In making this characterization, however, the Examiner has made 
numerous misinterpretations of Luck et al - 

For, example, in reference to step b) of Applicants 1 claim l, 
the Examiner cites to col, 7, lines 46-53 and col. 13, lines 17- 
23, of Luck et al . for comparison. The Examiner appears to have 
centered on the word "training" and assumed that all training of a 
neural network is equal. However, this is not the case. It is 
the specific method of training the neural network and the 
application of that trained network to actual samples that produce 
the advantages coming from use of the method of the invent ion _ 

In the method of the invention, training is based on 
selecting a first microscope image, using a number n of first 
image excerpts constituting a positive training set , the first 
image excerpts comprising the position of mass gravity centers of 
individual objects in the first microscopic image, and selecting 
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same 



and marking a number m of second image excerpts ("i n the 
first microscope image "), each spaced a predetermined minimal 
distance from the first image excerpt (sg_ that they will be away 
from individual objects ) , the number m of such marked second image 
excerpts constituting a negative training set . Then, training is 
made as follows: characteristic features (and/ or feature 
combinations) of the positive and negative training sets are 
determined, and a classification value between 0 and l is given to 
each on the basis of the positive training set being assigned the 
value 1 and the negative training set being assigned the value 0. 

For all further microscope images in the method of the 
invention, the system trained on the first imag e is then app lied . 
Classification values are determined and the recognition of the 
position of biological objects in these further images, step f ) , 
is based on a threshold criterion with respect Lo the 
classification values. Thus, in summary, the method of the 
invention is directed to automatic analysis of microscope images 
including training on a first image and recognition of the 
position (s) of biological objects in subsequent images following 
the training. 

In contrast, Luck et al . discloses classifying cells based on 
their morphology and not by recognition of their position (s) . 
According to Luck et al., a "primary" classification is carried 
out on images that have been found based on two scans. The first 
scan r at low resolution, is used to determine those areas of a 
slide that contain at least a portion of a specimen, A high 
resolution scan is then conducted on those determined portions, 
and the primary classification to find the centroids of biological 
objects having desired attributes is carried out (col. 3, line 55 
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- col. 4, line 19). Then, based on previously performed training, 
which is not described, with a training set of "several hundred or 
thousands of known benign and premalignant cells," the secondary 
classification is carried out to determine the the likelihood that 
the object is a specific type of cell (col, 13, lines 12-23). 
Thus, Luck et al . does not disclose an interrelationship between 
training, as recited in steps b) - d) of Applicants 1 claim 1, and 
use of the results of the training on second and subsequent 
microscopic images, as required by step d) of Aplicants' claim l. 

That the Examiner has mixed up several issues in Luck et al . 
in order to find some correspondence to the wording of Applicant's 
claim 1 is made more evident by the fact that steps c) and f) of 
the method according to claim 1 are not explicitly mentioned in 
the Examiner's analysis of the disclosure of Luck et al , In regard 
to the contents of Applicants' step c) , the Examiner states that 
the "complete training set consists of a positive (1) and a 
negative (0) training set" and makes reference to col. 13, lines 
40^48 of Luck et al . However, as described above, Luck et al. 
does not disclose two separate training sets determined on the 
same image , as required by Applicants' claim 1. 

Further, regarding item c) , the Examiner argues; 

Luck does not di sclose using minimum di st ance f or 
classification purposes. Computing distance for 
classifying images is extremely well-known as evidenced 
by Diouf et al - [sic, Watkin] see third paragraph on 
page 529. 

Therefore, it would have been obvious to one having 
ordinary skill in the art at the time of the invention 
to use a minimal distance for classification in the 
medical field (as disclosed by Diouf et al . [si.c, 
Watkin] ) in the method disclosed by Luck for no other 
reason than to classify the medical images correctly. 
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The Applicants submit that combining Diouf et al . with 
Luck et al- is merely impermissible hindsight reconstruction 
on the basis o£ the Applicants • claims. Furthermore, even a 
combination of these references would not result in the 
teachings stated by the Examiner. 

Diouf et al., at p. 25, third paragraph, discloses a formula 
providing a statistic that is dependent on the minimum of distance 
d wherein d is defined to be R' -mi', and wherein R 1 is random 
vector (see explanation of formula (7) in the right hand column of 
p. 528), and wherein mi is the expected value of a random variable 
(see once again p. 528, at the bottom). The distance referenced 
at that cite does not have anything to do with image excerpts. In 
fact, Diouf et al. does not do more than menti on the words 
"minimum distance." It appears that this reference has been found 
as result of a key word search including "minimum" and "distance." 

The Examiner then looks to Watanabe to obtain the other 
acknowledged "missing piece" from Luck et al., that the reference 
"does not disclose using l and 0 for training a neural network. " 
Watanabe, however, does not do anything more than to briefly 
mention values of 0 and 1 for the same variable (col. 5, lines 59- 
61) in the context of a "neuralnetwork and learning method." 
Applicants explicitly point out that the cited text passage in 
Watanabe relates to the same variable y, which can have two 
different values. It is clearly based on a hindsight approach for 
the Examiner to use Watanabe in this manner. in additioxv even if 
the references are combined, the combined disclosures still would 
not teach all the elements of the Applicants' claim 1, as recited 
above - 
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Finally, the Examiner makes no assertion that the combination 
of the cited references would even result in teachine Applicants' 
step f) and, thus, the Rxaminer does not even make a prima facie 
case for an obviousness rejection of the invention as particularly 
claimed by the Applicants. 

Thus, the Applicants submit that the rejection of claim 1 as 
obvious over Luck et al. in view of Diouf et al. and Watanabe has 
been overcome. The Examiner has, furthermore, rejected dependent 
claims 2, 3, 6-12 and 14 as obvious over these cited references - 
The Examiner submits that the indicated claims are allowable for 
being dependent on an allowable base claim, as discussed above. 
In regard to dependent claims 4, 5, 13 and 15, the Examiner has 
rejected them as obvious over the previously cited references and 
further in view of Hemstreet, III et al. The Applicants submit 
that the Hemstreet, III et al. disclosure in no way supplies the 
deficiencies in the primary references, as recited above, and, 
therefore, this rejection, too, has been overcome. 

Thus, the Applicants submit that all claims are in condition 
for allowance and such action is respectfully requested. 

For the Examiner's information, the Applicants have attached 
herewith two journal articles, published after the filing of the 
instant application, which describe practice of the method of the 
invention. 
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The Examiner is encouraged to telephone the undersigned 
attorney to discuss any matter that would expedite allowance of 
the present application . 



Respectfully submitted, 

TIM WILHELM WATTKEMPER ET AL. 



By: 

Holliday C. He/ne, Ph.D. 
Registration No. 34,346 
Attorney for Applicant (s) 

WE INGARTEN , SCHURGIN, 

QAGNEBIN & LEBOVICI LLP 
Ten Post Office Square 
Boston, MA 02109 
Telephone: (617) 542-2290 
Telecopier: (617) 451-0313 
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1 INTRODUCTION 



A Neural Classifier Enabling High-Throughput 

Topological Analysis of Lymphocytes in Tissue 

Sections 

Tim W, Natfckemper*, Helge Bitter', Walter Schubert*'* 



Abstract — A vernal cell detection system (NCDS) 
for the Somatic quantitation of fluorescent rym- 
phocvt«? in tissue a*ctUra* to printed. TJw *ya- 
t em Mtiuira visual knawl6dge frwn & net of training 
ceJMxcmse patches selected by a user. The trained 
system evaluates an image in two minutes calcu- 
lating (i) the number » (ii) the positions , and (iii) 
the phenotypeis of the fluorescent cells- Tor valida- 
tion, the NODS learning performance was tested by 
cross validation cm digitized images of tissue sec- 
tions obtained from inherently different fcypee of 
tissue: diagnostic tissue sections across the human 
tonejJ and across an inflammatory lymphocyte in- 
filtrate of the human skeletal muscle. The NCDS 
detection results Trare compared with detection re- 
sults from biomedical expert* and were visually 
evaluated by our moat experienced biomedical ex- 
pert. Although the micrographs were noi*y and the 
fluorescent cells v*xied h* ^bnpe and fliae, the NODS 
detected a minimum of 95% of the cells. In Con- 
trast, the cellular count? based on visual ceil recog* 
nit ion of the experts wore inconsistent and largely 
nnreproducxble for approx- 80% of the lymphocytes 
present in a visual Said. The data Indicate that the 
NCDS is rapid and deliver* highly reproducible re- 
eulte, and therefore enables high-throughput fcopo. 
logical screening of lymphocyte* xn many types of 
tissue* ae for example obtained by routine diagnos- 
tic biopsy proccdwaa. High-throughput nexrsroing 
with the NCDS provides the platform for the quan- 
titative analyse of the interrelationship between 
tissue OTviromnent, cellular phenotype and cellu- 
lar topology. 

Keywords' — Shape recognition, Object detection, 
Fluorescence Microscopy, Functional Proteomics 
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I. Introduction 

Tx; e h throughput screening (HTS) of lymphocytes 
in liquids samples by flow cytometry has been fcnown 
fat years as a powerful technique for rapid and ho- 
mogeneous measurements of many cellular parame- 
ters- The technique has been sitccessfully used to 
identity a laige number of lymphocyte phenotypes in 
thft blood and analyze their £unclionB. These studio 
are performed by the application of monoclonal anti- 
bodies (mAhs) directed against cell surface antigens, 
referred to as the cluster of differentiation (CD) anti- 
gens. Flow cytometric HTS has provided important 
insight into UTiinnnoregulatory events at the cellular 
level of circulating immune cells in many diseases, 
such as the autohnmune disorders, the immunodefi- 
ciency syndromes, and malignancies, and has led to 
numerous applications in diagnostic and therapeutic 
strategies ([IMa^W,^]). Many important aspiectn of 
the ralTi-ilnr immune system, however, are reflected by 
the ability of lymphocytes for homing and invasion 
of different organ Systems, where these cells organise 
themselves as dense lymphoid tissue, or simply mi- 
grate as single cells. This implies that the tissue spe- 
cific functional organisation of the lymphocytes that 
have escaped the circulation imist be analyzed at their 
coxrect anatomical sites. In contrast to the How cy- 
tometric measurements of lymphocytes in the blood 
or other liquids, the analysis of lymphocytes in tissue 
sections has remained restricted to visual evaluation 
in various types of microscopes fox years. 
The objective of this study is to introduce and validate 
a rapid learning algorithm enabling automatic lym- 
phocyte recognition and quantitative analysis oa the 
basis of digitized fluorescent images. The approach 
enables routine STS of lymphocytes at their correct 
anatomical sites. By its complementarity to flow cyto- 
metric screening of Lymphocytes in body liquids, out 
algorithm now allows the assessment of the functionai 
organisation of the lymphocytes in solid tissues by a 
highly standardised HTS procedure 
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Lymphocytes constitute an important fraction of the 
cellular Immune system- Brides their physiological 
role in lmroruioaurv«majace and defence of infections, 
the lymphocytes play an essential part in many die- 
eases, because they can invade and progressively do 
atioy organs. Fox example, relapsing lymphocyte infil- 
tration of target organs is the hsJJmark of the autolm- 
xoune diseases, and nearly any organ can be the target 
for lymphocyte infiltration. Besides autoimmune dis- 
eases, lymphocyte infiltration is the proximal cause of 
the rejection of transplanted organs* and lymphocyte 
invasion of the artery wall is an early, probably pri- 
mary event in the pathogenesis of atherosclerosis [Bj. 
Together these clinical syndromes conathrate the mar 
jor fraction of the diseases the western world. 
In the case of iofUtration of organs leading to chronic 
inflammation, H is a frequent routine procedure in 
clinical practice to asses a precise diagnosis on the 
basis of biopsies. Biopsies can easily be taken from 
nearly any organ ayfltem. Microscopic evaluation of 
lyrophocytea in these clinical samples is a standard 
procedure in pathology laboxatories. The classifica- 
tion □£ a lymphocyte infiltrate in biopsy tissue *sw» 
tions has "been largely restricts to visual nucroscopic 
analysis by medical experts. Usually lymphocytes in 
tissue sections axe stained for one gr«m surface anti- 
gen visualised by, for example, imioTmopcroxidasc re- 
action. Given many serial sections of a tissue site 
each stained for a different antigen, medical experts 
acquire rapidly a semi-quantitative result concerning 

i) the type of lymphocytes present m the tissue and 

ii) . the distribution of this type* However, many dif- 
ferent parameters of each single lymphocyte must be 
evaluated simultaneously. Correlating locally corra- 
sponding signals of fluorescent lymphocytes can not 
be done by simple visual inspection. Hence intercel- 
lular .functional correlations of higher order remain 
hidden in the image data. 

Fbr the latter purpose cell detection has to satisfy four 
main recjutoernentB: First, one microscope image has 
to be evaluated in a very short time to enable high 
throughput in screening many tissue sections. Sec- 
ond, results muat be reproducible to guarantee a rea- 
sonable statistical interpretation of the data- Third, 
a good detection performance by good generalisation 
is a requirement because cells vary in shape, density 
and fluorescence properties* Fourth, irncroecope iro- 
ag<p may be taken at different magnifications so that 
*he detection system must be easy to adapt to changes 
of the cells' pise) rise by a non-expert user. 
Detection algorithms in related applications arc baaed 



on the ideas of fitting a model to a gradient ensem- 
ble ([6], [7]), wave propagation [3], boundary tracing 
[9] or use the Hough- transform (p0],tll)) to 
circlo-lilce objects. In our present study the objects 
considered are lymphocytes during a migration in tie- 
true, so their shapes show considerable variation- Fur- 
thermore wc use immiuionuoxcscence labeling of these 
lvmphocytes. Tb» resulting images arc noisy because 
of heterogenous signals reflecting different concentra- 
tions of cell surface proteins which were labeled by 
»n antibody conjugated to a fluorescent dye. These 
factors mate it. difficult to define a model fitting for 
all cells in the image. Thus, many of the above ap- 
proaches seem unsuitable for our scenario. 
In this paper we present the Neural Cell Detection 
System (NCDS) which uses an adaptive neural clas- 
sifier, snapping each image point p in a fluorescence 
rmcrograph to an evidence value C{p) € [tyl]. The 
magnitude of the evidence value estimates the proba- 
bility for the image point to be occttpted by a fluores- 
cent cell. A local twa-rimum soarcb in the evidences of 
all image points combined with a thresholding proce- 
dure then can deliver the desired positions of fluores- 
cent cells. 

The clasBifier is a special kind of an artificial neural net 
which has been shown to be very efficient in fast learn- 
ing of non-linear classification tasks from even small 
traming sets. In tbi* work, the training aet consists 
of image patches from one fluorescence image. One 
subset of it contains positive cell classification exam- 
ples. These are obtained *ith the help of a human 
export who is experienced in analysing fluorescence 
micrographs. The expert identifies in the training im- 
age with a mouse pointer all fcheee locations where he 
judges a cell to be present- The other subset contains 
negative cell classification examples. Images rrom this 
subset are randomly chosen patches from cell-free re- 
gions- The purpose of these images is to adapt the 
classifier to the typical background structures that axe 
present in the microscope images and that must, be 
rejected by the classifier to obtain good performance. 
The training of the neural classifier and the following 
cell detection in a set of fluorescence images is done 
automaticany without uaer interaction. Thereby we 
reduce the required user interaction for adapting the 
NCDS to the selection of positive and negative exam- 
ples in a training set fox the neural classificator. 
-Far system validation of the NCDS the learning and 
classification performance was analyzed. The most 
important property of a ne\ixal classificator is its clas- 
sification performance on new data after training. To 
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measure the error rate of the latter we use ctoea -^li- 
dation. In addition, the cells, which were detected, m 
Images by the NCDS were compared with detection 
results of five human experts who marked the fluo- 
rescent cells manually and independently fro» each 
other. The resulting five expert Wste of cell posrtwwa 
were analyzed concerning the validity of these lists for 
calculating error rates. Additionally, the error roles 
of the NCDS were estimated by one expert, by visual 
inspection o£ its detection result- This * 
most outstanding experience m the vrtual analysis of 
fluorescence micrographs and te referred to as expert 
A in the following. 



p * - 



Fist. J- Fluorescence xwcrbgraph illustratmg CJ>V 
immanolabeled invasive lympliocytes in ran&de t™* 
The white axrowa point at some nuc-Toscen* Lymplic- 
cytea as examples of many others prevent fti th* site 
The cells vary m roiitnwt, ahap« and alae whieb is cliat- 
i^teristic for cells m otRan tissue structures. 



The objects for our cell recognition approach vwe 
nuoiescently labeled lymphocytes in tissue secfclona. 
The Ejections were obtained fcom various human tis- 
sues. Our analysis was focused on lymphocytes tev- 
ing invaded muscle tissue during disease (human sar- 
coid myopathy), and on lymphocyte* forming dense 
clusters within the tonsil. The tissue sections were 
processed for immunofluorescence labeling with mAbs 
antibodies against cell surface antigens as described In 
[12] and. [13]. We uaed a standardized pxotocol for the 
preparation of frozen tissues according to a) thictaiess 
of cryoatat sections (4 pm) and b) incubation w>th 
antibodies (concentration: l£J buffer (fl2|,p3|))- To 
label cell sairfaces of lymphocytes in situ, we have cho- 
sen mAbe gainst the receptors belonging to the clus- 
ter of difibxentiatiou antigens (CD)- Here we nave la- 
beled the CD4 and the CD3 antigens as characteristic- 
Visual fields within the labeled tissue sections were se- 
lected by a medical expert. The Huorescence signala 
of FITC (Fluorescdn-isc-thic-cyanate) cemjugated to 
the mAbs were vosualiaed in a standard fluorescence 
microscope (ZEISS Axiophot) equipped with a HBO- 
50 vapor lamp and a cooled CCD-camera (Photomet- 
ries MAC 300). Images were digitized under ^standard 
conditions including a) time for signal recording after 
excitation, andb) use of optical par aw tars (objective 
16* niter set BP 415/FT 510/BP 515-545 nm; 
Optuvat factor 1.25). The digitieod images were not 
post-processed. 

flignre 1 gives a characteristic example of lymphocytes 
of an inflammatory infiltrate In. the huma« muscle tis- 
sue, flnoKscently labeled for the CD4 cell sm&ce *nfa- 



m. METHODS 

In this chapter we describe the training and the ap- 
plication of the NCDS and the methods used for sys- 
tem validation, including comparison with cell recog- 
nition efficiency of five expert* (termed A^CJ^E). 



A. Thn Iteming Set 

lb train the neural classifier a training set T of M 
(input,outpnt)-pairs 

T = {(x tl) y ft )l,a-l..Af 

is created- The set is composed of two subsets T = 
V jxu ufn^s, The positive set = {(x5*\l]} con- 
sists of feature vectors computed (see B. below) 
from image patches centered at positive training sam- 
ples for fluorescent cells, together with the target out- 
put value yg" - 1 of the clasamcator. The other 
set P"* = {(x5*,0)} consist* of feature vectors x* 9 
computed from non-cell image patches (c£ B below) 
and their target classification output value y« = 0. 
The image patches for computing the feature vectors 
x£ w ^ 1*°* are obtained with The aid of an inter- 
active program that displays the digitised microscope 
Images and allows & user to select cell centers with the 
aid of a mouse cursor. Tbe set is then geaerated 
automatically by randomly selecting image paints m 
a minimum distance of pixels from any of the 
selected cells of T*". Tn this work we get r n „, = 5. 
Small changes of this parameter affect the system per- 
formance only marginally. For each of these randomly 
selected points a feature vector x£ flfl is computed by 
the same procedure as for r^. 
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. 2 a^Fluorescence micrograph ifluswattng mosde invasive lymphocytes labeled ft* CD 4 antigen at the cell flutfecft. 
Erllt^^^tSUphorytTdeTt.onBtr.iing their cellulw shape variation, c) W^l.s calculi from 
manual roouse-cll<*. Mlectian of 120 cross-sectfontd lymphocyte*. 



B. Calculation of Feature Vector* 

Choosing suitable features is crucial for the perfor- 
mance of a classification system. Id the context of 
our work the features should be robust against small 
changes of size* intensity and curvature of the fluo- 
rescent cells in the patches (see figure 2). Farther- 
more, they should guarantee a good separation be- 
tween cells aid xtf>n-ceBs in the tCin-dimensionsl fea- 
ture space. One way to map an image point p to 
a feature vector x <= lt*» is to calculate the over- 
laps of a surrounding image region of size JV x JV 
with a set of filters. Such filters can be a set of 
tuned Gabor filters ([U],[16]) or steerable derivatives 
of ^diroenaioji^ Gaussian* {16]- One disadvantage of 
these methods is that they contain several parame- 
ter {radius, orient ationB, etc.) which have to be fit- 



ted according to the particular application. Handling 
of nhis problem is difficult and tiine-cuxtauminjj iot a 
nnn-e^pert user. We use here a data-driven approach, 
a Principal Component Analysis (PCA) on a set of 
N x 7^-eiaed image patches of centered cells, ^ 
The PC A- technique uses dh% *igenvector9 v\ 6 M 
(so called "edeenceUs' 1 ) of the co variance matrix of 
JV x N image patches. The patches are the N x N- 
sized image regions around the hand-selected image 
points of fluorescent cells- Figure 2 gives a set of 
patches (2b) with the eigencells corresponding to the 
highest eigenvalues (2c). The usage of such eigen- 
vectors is a vfelHmown technique for detection tasks 
([17], [18]). A. listing of the eigenvalues in descending 
order reveals that majority of the variance in the im- 
age data is described by the six eigencells of thn six 
highest eigenvalues. These are taken to generate the 
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e^dimenaional feature vector x for an image point by 
scalar nmltiplicfition of its N" x N neighborhood with 
the six eige»ceJJs. 

C. The Newel Classifier 

We use an arfciflcal neural net of Local linear Map- 
type (LLM) fox classification. The LLM was intro- 
duced hx [19] and has been shown to be a powerful 
tool for the fest leamrng of non-linear mappings 



Biicb «ls classification tasks m ComputeT Vision appli- 
cations [20]. The LLM-appro*ch was wiginauy moti- 
vated by the Kohonen ScM-organiadng Map ([21U22]) 
with the aim to obtain a good map resolution even 
Tirith a small munber of units. Recently, there have 
been a number of proposals for algorithms with the 
game objective, but a somewhat different mathemat- 
ical basis- The GTM-approach [23] creates a map 
majdifbld on the basks of a statistical Estimation pro- 
cedure (mardmiim likelihood approach). While the 
authors argue that their method, therefore, is "more 
principled" as compared to the standard SOM, it is at 
the same time also computationally heavier and more 
.complex to implement. Another proposal is by [24], 
ivhich la rather attractive due to its high computa- 
tional efficiency- Other proposals, such ae [25], com- 
bine the SOM with the (oomputationalty also much 
more expensive) Sammon Map [26], using anb H am- 
pling strategies and MLP-ba*ed interpolation schemes 
bo arrive at a computationally efficient method. Since 
tre here are primarily interested in classification, the 
present application is unlikely to benefit much from 
the modifications proposed in the cited references. Eor 
a quantitative comparison with some more standard 
approaches, see e.g. [27]. A collection of recent appli- 
cations of the SOM can be fcund in [28], 
In the present paper the dimensions are -— 6 and 
= 1- In the following the algorithm 13 described 
m general The LLM combines unsupervised and su- 
pervised learning in contrast to the widely used multi- 
layer perception trained with back-propagation [29]. 
The LLM is given through 

{wj» e JrVKw?" 4 S TR*" 1 , *i e M*"*** 4 ,* = l»n}. 

Wc call a triple v* = (w^.vrf*, A 4 ) a We. Once 
the LLM is trained an input vector ac ia mapped to an 
output <7(x) by calcinating 



with « = arg min{|l * - wf So ™ the newest 
neighbor to input x. The mapping into the output 
space is performed by a local linear transformation 
given by A* and vo*"'. An illnfibiation is given in fig- 
ure 3. 

lb train the three free parameters of each of the 
n nodes (wf , wf\ A„i = wc (5) Relect input- 
output pairs (xaja) from the tTainiug set T = 
{(x^y^)}, (ij) find for each the best-match node *, 
»d (ni) change its weights according to the learning 
rules 



A A a - ^(y* - CM) 



(3) 



with ^ ft , e^S e* €10,1 [ as Learnine step siaes. Looking 
at. the rules, one can observe that learning rule (1) is 
the online version of k~roeans [30] for positioning the 
n centers of wf 1 (gee figure 4). The learning ru.es (2) 
arid (3) are for the adjustment of a linear mapping 
specified by vector w™* and matrix A* in the vicinity 
(Voioncd cell) around the best match node. 




Input space 

Fig. 3* Example iDustrating a Local Linear Map approx- 
imating a function €= 1R 3 (dotted plane) with 
five nodes. The lXM's nodes' wj rt form Vnronoi cells 
of the input Space. The plane is approximated by pro- 
jections of the cefle (on* sketched as dark giey plane). 
An input vector x U mapped to C(x) In Ihe output 
space, first, the neareat aeigbbot w{» to tins inpnt 
is selected, then the Input is mapped via the coupled 
matrbc A H (dashed arrows). 
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D- Automatic Cell Detection 

Tb calculate the desired positions of the fluorescent 
ocUb in an image every image point p is mapped to 
its evidence value [0; X] by calculating the feature vac- 
tor x of its surrounding N x -flMmage patch and its 
LLM output C(x) € [□, 1]- The output is the evidence 
value for a fluorescent lymphocyte at position p. By 
mapping all the evidences to their corresponding im- 
age positions we achieve a bo called evidence map. A 
point p m the evidence map which has the highest 
evidence value C(x) uH*hin a ^ radius above * give:* 
threshold is the result position of the center of a flu- 
orescent cell, if C(x) obeys 

Otv) > 0-6A<7(p) =* orgriiBx{C(p0}< 

S. Optimized Learning with Activity Equalization 
Vector Quantization 
One problem of a winner-take- all vector quantisa- 
tion in equation (1) of the feaxniag rules b thai the in* 
put feature space may not he approximated efficiently- 
The reason is that a fraction of the randomly initial- 
ized wf 1 can become stuck in regions of the input 
feature apace, containing none of the x*. lb achieve 
a better quantization of the input space an extension 
is made to the LAM learning scheme, the so catted 4c- 
tivity JSqualiza&an Vector QutMnza&m (ABV) which 



<*sr* «w 



-waa introduced And discussaed in |S1] and [32]. After an 
epoch of T learning steps for each node we evaluate Hg 
activity. The activity of a node is> defined as how often 
the node was the nearest node to a training input. We 
define the average activity <h of the «-th node xrithin 
a sweep of T learning steps as o« = r £fcf*T l 

with 




KwT^(*)) = l 



1 tf wj* is the nearest neighbour 
to the input at training 
step t e [1...T), 

0 else 



Fig, 4. The first three compcuieats of the 6-d*m«a*io»al 
trauung vectors *« are plotted with an ABV quan- 
tisation by 25 referew.ee vectors wj 71 (grey-dark grey 
bullets) for example. One can observe that the cell 
fiivbures (grey plus-symbol) and the non-cell features 
(bliLck diamond) form two overlapping clustery The 
ofclier three components the components have a slmlte 
distribution, All 25 nodes contribute to the approxfan*- 
tton of the feature distribution to the same amount and 
a. goad approximation of the distributdoti ia achieved. 



x(*) denotes the training input of the *-th training 
step of a period of T steps. The boaic idox of the AEV 
ia that after each T learning steps the activities of the 
nodes axe computed and a set 

I « < 0} 

of "inactive nodes" (i.e^nodeB whose average activ- 
ity is below a chosen threshold &) is determined, A 
randomly chosen fraction / of nodes in J then repo- 
sitioned by shifting them by » minimrd distance TmmpL 
in a random direction, ie. the new node has to obey 

\\v>f^ - Tempi- 

The rationale behind this is to ensure a better explo- 
ration of the input space- 

The remaining fraction 1 - / of nodes in J ia repo- 
sitioned to random locations in the neighborhood of 
active nodes 

I, e., the new positions 1^"™ of these nodes have to 
obey 

The rationale behind, these moves is to ensure a bet- 
ter activity equalization among nodes and to better 
populate densely chiatered areas xa. the iwput space. 
For the application of the AEV the Ti,^ learning 
steps are divided into three intervals [1, . - > ,Tx] f [T% -r 
l >1 ,- 1 T a J-Bdpk + l l ... f D < «m]. ***** ph**e 
after every T learning steps the sictivities of this nodes 
are computed and / is chosen as / = 0.5. After the 
first phase the exploration is finished and in the sec- 
ond phase wo chose / = 0.0, i.e- reposition nodes 
only. In the third phase the AEV is finished and tbe 
LLtM is trained by wplvmg teaming rules 
Jn this supplication the LLM is trained in T{ W n — 
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n * 10 5 steps- The interval borders -were act to T L — 
0-2 * and T 2 = 0-4 * T lBwrn . The length of a 

period T was set to T n * 100. 



i?. ^5tem Validation 

Fbr system validation the performance of the NCDS 
was analyzed concerning two aspects. First, the lean* 
mg and generalization performance of the LLM was 
analysed by cross validation [33]. lb perform a cross 
validation the data set T that we use for oirr evaluation 
is divided into six disjoint subsets T = \J % r Sl * = 1, 6 
each of eojial s^ fi ' containing the same numbers of 
positive and negative training samples. FiVft of 
subsets form the training set IW* the sixth is 
the test set rw After the NCDS haa been trained on 
rV* n the inputs from r«e«t ax« classified to evidence 
values by the LLM. Because the test data consists 
of feature vector* computed frit angle image patches, 
only the classification performance of the LLM on the 
feature vectors ib evaluated. So only the evidence val- 
ues calculated for the feature vectors of Tut* axe con- 
sidered here. If the evidence value x« € rw, 
is larger than 0-5 tho patch is classified as a cell, oth- 
erwise It is rejected. Thereby a classification rate for 
the test samples of was calculated fox the ^posV 
tive camples and for the negative ones, r teJ V We 
then repeat the entire procedure fire more times with 
each Ti selected as IW Classification rates on neg- 
ative and positive examples are calculated separately, 
Tho rates give a good approximation shout the classi- 
6V>rs ability not only to learn the training set but also 
to classify new input data correctly. 
The second aspect of validation is to apply the NODS 
to the images, and then measure error rates for the 
detected positions of fluorescent cells by Beceivet Op- 
erator Characteristic (HOC) measures (eee [34] for a 
review). One error measurement was done by visual 
inspection of the NCDS^tectod cells by out most ex- 
perienced expert A, In addition, the NOD S detection 
result was also compared with the results of cell de- 
tection, carried out by human mdividualfl, For this 
purpose one preselected micrograph was evaluated by 
the NCDS and by expert A and four more experts 
vith less experience than expert A (called B i O,D 1 E) 
independently. The experts denoted the positions of 
fluorescent colls by mouse clicking at their centers. 
Thereby bU independent lists of cell positions ware Ob- 
tained, lb calculate error rates of these detectjonre- 
*uts list was selected far the "gold standard" . We 
used three different gold standards to evaluate both 
the NCDS'a and the expert's performance: 



a) The detection result of expert A was selected as 
the gold standard because of his outstanding expe- 
rience in interpreting fluorescence images. 

b) The detection result of expert B who has a medium 
image evaluation experience was selected as the 
gold standard. 

c) By merging the five lists of a)l experts into a new 
list we simulated a sbrth expert ("OR^pcrt"). 
Only those cell positions were written to this list 
which were included in mote than one expert list 
so that at least two experts had consensus about 
this fluorescent celL Identical cells in different ex- 
pert lists wero found by a local nearest neighbor 
search m the positions. If the Euclidean distance' 
of the two cells at p< and p, was less than 5 pixels 
the two positions were merged to one position at 
fa + pj-)/2 in the OR^expert list- 

Ebltowing [35], two measures were calculated to char- 
acterise detection performance based on out gpld 
standard for the human experts and fox our system. 
SS: The sensitivity (SS) of a detector is the percent- 
age of cells of the gold standard which were found 
by the detector. 
PP V: The positive predictive value (PPV) is the per- 
centage of detected cells which were also included 
in the gold standard. 
Both SS and PPV were calculated for each human 
detection and for the NCDS's detection using the gold 
standards a) and b). 



IV. Results 

We have analysed human cryostat tissue sections 
from inflammatory skeletal muscle tissue and from 
the tonsil for presence of lymphocytes fluorcscently 
labeled for cell smfece antigens. Immunofluorescence 
features of the lymphocytes were inherently different 
in these tissue types. In the inflamed skeletal muscle 
arrays of lymphocytes were accumulating within the 
connective tiasuf! ant rounding individual muscle fibers 
(figure 6a). The latter remained free of any fluores- 
cence label, whilas the fluorescence signals *-ere re- 
stricted to the lymphocytes surfaces, indicating speci- 
ficity of the labeling procedure ([^,[131). By contrast 
in a cross-sectioned T-celi area of the tonsil, densely 
clustering lymphocytes were seen (figure 6b>. 
Muscle tissue- and tonsil tissue-sectiona, both labeled 
fat CED4 cell surface antigens, were subjected to our 
teaming algorithm at low and high modulations, re- 
spectively. The magnification for muscle sections was 
97x (figure &a), 242* for the tonsil (figure 6b) 

ui order to examine the influence of cell magnification 
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TABLE I 

Analysis of the detection resvlts op thr pour 

HUMAN (B-F) AND THB ISTCDS BY 

COMPARISON TO THE RESULT OF EXTENT A. 



TABLE II 

ANALYSIS OP THB JOaTJSCTlOTT RESULTS OF TUB FOCR 
HUMAN fiXPEWTS (A^'-K) AND T»B NODS RY 
COMPARISON TO THE BBSULT OP EXPECT B. 





B 


C 


D 


iE 


NCDS 


" S3 
PPV 


0.73 
0.45 


0.70 
0.53 


0.78 
0.60 


G.20 
0,03 


0,97 
0.26 





A 


C 


i D 


E 


NCDS 


S3 
PPV 


0.51 
0.74 


0.51 
0.68 


0,51 
0.63 


0.13 
0,92 


0.96 
0.32 



on the validity of the NCDS, 

The results of the cross validation are shown in fig- 
ure 5 (muscle, figure 5a; tonsil, figure 5b) 

The Myograms indicate that 00 - 95% of the pos- 
itive training samples of iWtest = 1 — 6) axe clas- 
sified correctly for both tissue types. The rate* of the 
correctly rejected negative samples were 80- 87%. Aa 
indicated in figure 5 there is considerably low intxa- 
aud iufcer-individiial variation of classification rates, 
lb compare the expert's and the NCDS's detection 
performances, the NCDS was trailed with a given 
data Set obtained by maamal selection of cells car- 
ried out by expert A- The NCDS-detected positions 
of fluorescent cells were- denoted by white boxes in the 



TABLE III 

Analysis op the detection results of thb sons 
human BxPfiata (A~E) AWD NODS in comparison to 

A SIMULATED OR-EXPEKT. 





A 


B 


C 


D 


33 


NCDS 


S5 
PPV 


0.70 
0.92 


0.80 
0.65 


0-78 
0.84 


0.80 
0.S0 


0.1ft 
0.QG 


1.00 

o.aa 




Pi K S. Tbe histograms representing the el&Ralficarion rfttcs 
of tK E NODS of the she test sets I\ for the 
vadhig ra) and dusking lymphocytes in the tonsil (b). 
The black boxes axe the classiflcation rates far the pos- 
itWe test RampJes (true positWes) while the grey ones 
stand for the rate <rf the negative samples (trtt* nes*- 
tives) for each toart set iy 



imagce (see figure 6e and 6f)- The positions of the de- 
tected cells were carefully inspected by expert A and 
to hia evaluation result the PPV of the NCDS was 
estimated to be greater than 9056 and the SS to be 
95%. 

The results of the comparison of experts and MCDS 
to gold standards a),b) and c) are shown in Table 1, II 
and HI. Although the detection Tea Lilts of experts A 
and B show considerable variation (see firs* column* 
of 1 and JJ) the classifier trained with expert A's data 
shews very good generalisation performance in reach- 
ing similar SS and PPV values for gold standards a) 
and b). In a next Step we aimed at the evaluation 
of inter-mdividuaX detection reproducibility of visual 
cell detection on a screen showing intensity images- 
Abo it is shown hi TVvble I and II there is a consid- 
erable variation in tne detection results between all 
individual experts indicating low reproducibility (for 
xaore details see table captions and the Discussion V). 
For a more objective study we constructed a new gold 
standard ( fa OR-e*pert" ) containing all cells detected 
by at least two experts (c£ section HI-F). The result- 
ing table III indicates that agaii^ there is substantial 
inter-individual variation of cellular detection- Ap- 
proximately only 20% of all cells were reproducibly 
recognized by ail experts. To complete the series of 
e^wimants we have used the algorithm fi>r CD4 sig- 
nal recognition to detect cells iromunonuoreecently la- 
beled for CD 3 (see figure 7a). As shown, in figiare 7b it 
is readily seen that the number of correctly recognized 
cells is above 90% as substantiated by direct evalua- 
tion of expwt A. 

Hbr a 6&8 * Sl7-tirc& image with patch si&e iv = 15 
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. e. Three ateps of ceU ^^^^^^ 
.Wfl the two input images nee '^^^l^ZT^^SS^m^ the «Tl. wero immur-nlafa^ again* 
lymphocyte, for«iT, H ^ ^ tffmlieto £^ 55 *• clustering cell* to the tupstt (d) 

CD4. 1~ ^^drow tb- Sn^SSSS^^i claerifier were mapped u. grey vnlu*, A. 

are presented. The -video** vajmss QU) € WJ^™*^ 3 ^ wUto ^ m the comers show enlarge, r^ons 
high 5tends ft* a high ^^tj^^^^'^l^X otf«^£* that high evinces coxr«pond 
m»*e and evidence ™*P H^^L^S^TShSs ddiv^ the <WJ cdl Prions ^thto 
to positions of fluorescent cells. Lodal minimum analysa* ana tnreanaKHms u 
whit* frames, for both muscle tisso* (e) and to™a (f). 
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as a rough estimate of cells ma^mum ; diameter the 
training of lie LLM takes one minute; the process- 
ing of an image takes two minutes on aj standard PC 
equipped with an AMD KG/300 processbr with 96MB 
RAM- The number of nodes of the LljM was set to 
n = 2B because the LLM showed bests classification 
performance in this case. j 

V. £>I£CUBSlON 

In the present paper we have described a fast and 
easy to handle algorithm for rapid detection of fluo- 
rescent lymphocytes in tissue sections. The detection 
time of a visual field is approximate^ two minutes 
enables the evaluation of at least cells within 

throe hours. Tho perfarmancc speed of the NODS 
grows quadxatically with the neighborhood aiasa N , 
so that an image is anaryj&ed in 0(N?) steps. The 
most time^onsumhig operation is tie calculation of 
ihe feature vectors. Because this is Acme by scalar 
multiplication, a speed-up can he obtained by spe- 
cialised hardware. The evaluation Of one image in- 
cludes eixxrtdtaueously (i) the number if cells, (ii) the 
molecular phenotype of throe cells » arid (iii) the pre- 
cise positions of these cells in the ^y-dlrection in 
given tissue sections. Thia direct totiolo^cal high- 
throiighput-approach enable* us to investigate hidden 
relationships between structure, molectilar expression 
and function- Pbr example, in polyrnytotis, a human 
inflammatory disease, it has been aho^ki that invasive 
T-lymphocytes migrating in the leading front of an 
army of invasive T-lymphocy tea express a cbU surface 
phenotype, which is specific for these Want T-cells [13]. 
T-cells behind thia invasive front eacprees substantially 
different receptor proteins at their cell surface. This 
indicates clearly that there is a Btrict dependency be- 
tween the relative positions of pathogenic T-cells and 
the phenotypes expressed by them [13J, Similar inler- 
ralationsbips are observed in secondary immune or- 
gans such as the tonsil, where T^yknphocytea and 
B-Oympuocytes expressing different ptotein receptors 
(CD-antlgens) at the cell surface are jnffie^ntially oi> 
deied into specified tissue areas, referred to as the 
T-ce]> and the B-cell-areas [S6]- Thek two examples 
show that dependencies between strufcture, molecular 
expression and function can be virtahtecd by fluores- 
cence microacopy. However, it mmi be considered, 
that at. present we are not aware of | most of the re- 
lationships between tissue environmejnt, multiceUular 
molecular phenotype and topology. This is clearly the 
case in inflammatory infiltrates which axe diffusely 
distributed in the tissue such as in' our pnBext ex- 



ample of sarcoid myopathy, A systematic analysis 
Would requirn highly standardised visiu*Ai*atjon proto- 
cols and topological analysis of high reproduribifcty to 
generate valid multi-dimensional data-spaces. Ii* our 
present paper we provide an appropriate system for 
the latter processes ft* enabling efficient data-mining 
and visualisation. 

Several lines of evidence indicate that the NCDS algo- 
rithm works with high precision in individually struc- 
tured tissues and builds en essential part for extrac- 
tion of reproducible data from fluorescence niicro- 
graphs. First, the results of the cross validation show 
that the neural net performs a satisfactory generaliza- 
tion on the data- The system's classification rates re- 
main constant under variation of training-data. Una 
is an important feature of the NODS, because in prac- 
tice the training sets collected for adaptation are gen- 
erated from different users, so that the sets show vari- 
ation, The small variations between the classification 
rates show that the NCDS 1 * detection performance 
remains constant against these variations. The small 
variation between rates calculated for training sets of 
the lymphocytes invading muscle tissue and cluster- 
ing within the tonsil show that the NCDS also deliv- 
ers data of constant accuracy under variation of scales 
considered h&xe» 

The lower classification rates on negate samples in 
both image ensembles show that the LLIVTs classifica- 
tion i$ more sensitive than the experts - 
Second, we demonstrate that image evaluation by ex- 
perts show high variation indicating low reproducibil- 
ity. Therefore automation Of the detection process ifi 
necessary for valid statistical analysis. The devotion 
performances of experts A-E are discussed briefly m 
the following. Loolrins at table 1, it is surprising tha* 
the values for SS and/or PPV with A being selected 
for gold standard are rather low- The results can be 
divided into three groups- Experts B, C and D show 
similar SS-PPV proportions although Trith high vari- 
ances. Expert K achieves the best rate In detecting 
cells of the gold standard (93%), but at the expense 
of a very ^conservative* decision behavior he misses 
60% of the gold standard cfille- The third group is 
the NCDS which has the best SS-rato but seems to 
pay the price for this by having a low PPV- Table 
111 shows a rfwHw pattern of detection performances. 
It is interesting to observe that the PPV rates in- 
crease whik the SS-ratas rrvualy remain constant for 
all human detectors in relation to table L Thia indi- 
cates tfcah many cells that were detected by more than 
one expert were missed by others. This clearly shows 
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-that none of the human detectors is valid for a gold 
standard. Even a simulated OR-expext provides no 
further improvement, because it leads to the setting 
of new parameters like the nearest ndghbor-distance. 
The reason for these unrepTaducible reetdts obtained 
by the experts ifi that visual evaluation of grey value 
images is very tiresome leading to significant decrease 
of visual attention even within tea minutes of woris. 
Third, visual inspection of the NCDS'h detection re- 
smlta led us to much better performance vahieB of SS 
and PPV for the system than for any of the expert*. 
The difference to the SS- and PFV-values which were 
based on comparison with both gold standards is not 
surprising here, because it underlineB the inaccuracy 
of thittiati evaluation used for the gold standard. 
The highly inaccurate results obtained by human ex- 
perts in relation to the demonstrated reproducibility 
of our system shows that the NODS can pave the way 
towards a more systematic analysis of cellular systems 
in their Correct tissue specific environment. 
Since the results of the NCDS show high accuracy 
and reproducibility the system is established is. rou- 
tine work in our Lab and its efficiency is atfttrotlad i u 
a closed loop between development and application. 
Because our work aims at the development and im- 
provement of an evaluation tool for a particular image 
domain, fluorescence micrographs of tissue flections, 
it is very difficult to compare the algorithm with ap- 
pxoaches in related domains as mentioned in chapter 
I. Because the fluorescence imaged contain very much 
noise and the shapes and sizes of the lymphocytes 
show very much variation the algorithms are definitely 
not applicable or must be timed to such a great extent 
that a comparison would be out of the scope of this 
article. 

Given the cellular immune system the lymphocytes 
we organized in different compartments including sec- 
ondary Immune organs and the blood. Hitherto, 
large-scale cellular phenotype analysis has been re- 
stricted to flew cytometry automatic quantitation of 
lymphocytes in the blood ox other types of liquids. 
We now provide a tool to extend automatic quantita- 
tive lymphocyte analysis to the tissue compartments 
of the immune system, 
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Automatic Recognition of Muscle-invasive T-lymphocytes 
Expressing Dipeptidyl-peptidase IV (CD26) and Analysis of the 
Associated Cell Surface Phenotypes 

V. SCHUBEST'*'*. M- FMEDENBERCER'. R- HAARS", M. BODE 8 - L. FHJIJPSEN*. T. NATTKEMFER' and H. RTTTER? 

' gimp. Unfvrrsily «f Bitlifdd. 33619 BitUfari. OtnnW 

(Receive I Aapui 2000: In fmafnrm 23 April 2O0I ) 

A nc^rul coll doueU™ wttem (NCOS) for the qu-ntUffli™ [^S^^IrrffiSlS 

tissue section* *m used » analyze CD26 wpremaon in mOselo-invaaWo T-onTls- Cn26 I s a ccU sui&ct 
^Sy?-S^» IV CPW IV) ^ivcd in oo-KtimulBt^y a£tiu a d<» nf T-edte and almo in adhadvc 
cS^eM^sy**! acquire* vtoal knowledge tarn a f«,<> r ««»'"£ c ^^^ s "™ 
by a user. The crd-ed system evaluates an image in Zmta calralatms « «w ^-00^^ 
and 0^0 the phenotypes of the fluorescent eells. In the prcsem study we tavc u«cd the NCDS i ^™ry 
5PP IV (CRM) a&np invasive lymphocytes in sarcoid myopathy and * JftSSSEn 
cell suffice rhcnOtvW ^ find M Ehhr unusual phenotypes character^ by drffei»rt»al eon^manon 

suooort b differential adhesive rather than a oo-sdmulatory nd» oF CD26 u» m^lo-mvauve colls. The 
^SiliS S STNCDS algorithm to divert typos of adU should onahla unto approach any -voskk, 
pro corp, including invasion of malignant cell?. 

Kerncrdf. Muscle; CD 26: Neural network; Image analysis; InBanmMiory myopathy 

1 TKTornmrTTn™ proximal tubuli of kidneys (Stein etol.. 1989; Hegsn e»oi. 

1. JNTRODUCraW ^ Bfihlhlg ^ ^ l995> _ r^pp tv Is implicated In 

D^ntidYl-pcptioW IV (DPP JV, CD#i) is a cell inflammatory procew™ and appear* » Pj»y «" ** 

SS^nSSn. protein dMMd by a she pn^ion of «n*» ™^ £^ Ll£ 
N^idd cytoplasmic domain and a long extra cellular 1993; Ivrflia and Monmoto. 1999) Chveb j*e ccMubar 

^oTwfiTSS rich and a Cymric*, d^a*. A * W i™-^^^ Tv If"" s f " 

260 nimnoacid Otemtinol e«ra cellular region vraa .rrrportant funcfwi of DPP IV is 'ts role as a cc- 

Jclrr^rSpP TV erzymc eatarytio activity. cel. T^ 0 ^T «" ,K mV0 ^ * * 

(Henencini 1990; Darmoul « oi. 1992; Tanaka « al, acnvation of the T^ymphticyw. , nrtae «. 

QQ^TnPP IV is a member of the prolyl oBeopeptidase Within the T-cell activation cascade, antibody -induced 

ZelT s dc&S by the Wntanwrf the stimulation of DPP JY leads to tyrosine phosphoiytation 

^nwTv^Ko shown tot* expn,,BCd by a variety of eon,pl«. CD26 provides a true co-Mirrjutotory ftnicHon 
cells, and by eptthcHa of the Intestine, the prostate, and the AeTCR. 
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TABLE I CP ocfl mtyfacft antigen analyzed in Hri« study ^ 

-— " — ^ Mnnockina l anribnd y clone 

CP an tigen . spacing . ' " " 

■ ■— .-. ■■ L272 

en m DPP TV: adhesive deaminase binding p?c«*" RQ . . 

55 f Cft-r«MBnlli« iwbpU*«»MHC oi™ T wi* TOR 

Pnl co-rect.gftM«i reCopior for MHC 4us II «4th TCR !^ 

mllb aM l««ffrin chain of M/\d wmplo*, Obi n*eplmJLR3 *f 

CD Si ***** of <h= Sl B ^l^'™ ^'^^ 39C1.5 

X, 3 Signal errodpewn nttcptar complex ampcrated with TCR _ ^_ ■ 



In pcriphOTsJ blood the T-lyraphocytes that express OTP 
IV are dther CD4 or CD8 positive cells* and. a? a xule 
show compression Of intact TCR^CD3 compJewe*. Hen« 
a T-ccll chat can be co-stimulated via CD26 is a 
CD4 + /CD3 + /TCR + /CD26*. or a 

CX>8^CD3*/TC^t. T-cel L Tn addition, the majority 

of these T-cells also sprees CD 2, which can provide an 
attematrve pathway of T-cell aenvafion (Davis rt al*> 

199S). ^ ^ . 

In the present paper wc have addraftsed the expression 
of DPP IV and DPP TV-associated cell surface phcnotypcB 
in muscJe-invasiveT-lyinphcicytos. Multi-epitope Imaging 
necroscopy (Schubert. 1992) wns used to co-localize 
seven different cell surface proteins including CD26 on 
the cell surface of these muscJe-mvasivc T-tells. Tha latter 
technology was coupled co a. new learning algorithtn 
(Nanketnpcr «f al> 1999, 2000a), which amflmatically 
recognizes T-oclls within tissue sections obtained from 
patients suffering from chrome inflammatory ranscte 
disease. We have selected chronic sarcoid myopathy, 
which represents a human disease type showing T* 
lymphocyte invasion of the rouncjc tissue. The T> 
lympbwytee are present to a larg<* extent within the 
connective tissue between the muscle fiber fascicles, a 
space that is defined a* the perimysinm. in addition, 
Roreoid myopathy is chataeujifisted by the Rtmarion of 
giant cells, that is supposed to bo driven Vy * e 
cooperation between T-cells and macrophages <yf*h\- 
strom eta\„ 1999), 

In the present study we have examined whether muscle- 
invasive T-lymphocytos exhibit a cell surface phenotype 
required for T-cel) activation and CD26-aRsocdaced T-cell 
stimulation. Using a library of seven different monoclonal 
antibodies we find that the majority of the T-cells do not 
co-express the cell surface receptor sets, which would be 
required for T-cell activation via the co-stimulator 
molecule CD26. Insicad these T~ceJls express unusual 
cell surface phenotypes by heterogeneous receptor 
combinations, most of which are minus-variant* of the 
phenocypes found in the blood- We suggest that these 
"unusual" cell surface phenotypes are involved in 
differential adhesion m«Aairi*m5 and T-cell migration 
rather than T-cell activation. ^, v _ 

We also describe a learning algorithm, by which T-ceUS 
wfthhi the tf&Rue can be automatically ^cognized and 
quantified. Given chat a Urge number of tissue sections 



have to be mapped by multi-epilopc imaging, trie 
algorithm opens the possibility for high-throughput 
screening of invasive lymphocytes in tissues (Naitkemper 
eial* 2000a). The adaptability of tlie algorithm ro diverse 
types of cells shouW enable us to approach any invasion 
process, including invasion of malignant cells. 



X IMAGING METHODS 

In order to address the CTOG-a&ftDciated pbeootype of 
muscle-invasive T-cells. we have applied seven mono- 
clonal antibodies directed against cell surface antigens 
(CD antigens) in cryosections of diagnostic biopsies. The 
antigens are listed in Table L AH antibodies were directly 
conjugated to dyes and applied to the tissue sections as 
described earlier (Schubert, 1 992). Each fluorescent signal 
was recorded as a digitized image by a cooled CCD 
camera. Fluorescent cells were either localized by medical 
experts a mouse-delineation of the fluorescent area, 
or were automatically recognized by the leamfng 
algorithm described below. 



X AUTOMATIC T-CELL FLUORESCENCE 
PATTERN RECOGNITION 

To identify CD2G positive T-cells and other T-cell types in 
tissue sections we used a modular computer system that 
detects the positions of up to 95% of all fluotcscent 
lymphocytes in one given input Image M, the digitized 
fluorescence micrpsraph (Nattkempcr et aU 1999), The 
first module of mc ceJJ detection system is a trained call 
classifier that classifies a square image region p of 
] 5 x I S pixels to a so called evidence value C(p) £ [0\ 1 ] 
^presenting the probabiKry that the centei of occupied 
by a fluorescent cell. The second module evaluates the 
evidence values of all pomW m M to a list of positions of 
fluorescent lymphocytes. Tho positions of the detected 
cells are visualized on a screen and stored in a databass. In 
die following subsections we describe the cell detection 
system briefly, see (Narfkcrnper et ol. 1999, 2000b) for 
details. 
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is called node, In the present work the X-LM parameters 
By calculating 

C(x) = wj"* -h x\ A (x — >vj*} 

the input feature vector * is mapped to the evidence value 
C(x)-fc holds 

fc = fl 4*mm{|[x~vJ/'||}, 

bo Hijn is the nearest neighbor to input x. An lltostration is 
given In Fig. 1. j 
The throe free parameter* of each of the n nodes 

trained with a pte-*elecicd sec rot m flnpui, cwrput}- 



arc 
pahs. 



4. TRAINING OF THE CEUL CLASSIFIED 

The cell cUsrafier CQp) m a trained artifice neural 
network (ANN). Neural networks have been shown to he 
powerful classification tools \n many industrial computer 
vision applications In blornftdical image analysis the 
application of ANNs is not frequent, and only reoaidy 
applications have been published (Sjoestroem et al. 
1999). 

To map an image region p to its evidence value, hx 
numerical feature values arc calculated for p and 
combined to a so caned feature vector x € /K°. Here the 
term "vector" describes a ser of numerical elements as 
used in the field of computational pattern recognition. The 
computation of the feature vector is described la 12. The 
trained ANN computes the evidence value for p by 
mapping to feature vector x to COO ubJih* th* learned 
classification mappings C : /R 6 ■ JR. 

The ANN used for cell classification L* of Local Linear 
Map-type 0-LM) which was Introduced by Ritter (1991) 
and has been shown to be a powerful tool in faa Icwtnln* 
of non-Hnear mappings, 

such as clasinlicarJon tanks In Computer VisJanappK- 
catinns (Heidemarm andRitter, 1 999). The t^M^pptxach 
was originally motivated by the Kohoncn Self-orsanmnB 
Map (Kohonen, 1989) with tho aim to obtain a gocdmap 
resolution even with a small number of ante. In Ac 1AM 
learning scheme unsupervised and supervised learning a™ 
combined in contrast to me widely used nwla-kyeT 
perception trained with back-propa^tlon (frummelhan 
et qL, 1986). The LLM Is given through 



that is competed of two subsets r = U I™*, TOe 50 
called positive set 

consists of feature vectors jr£* computed (see below) 
from Imago patches centered at positive training 
samples for fluorescent cells, together wilh the target 
output value 

of the classifier, Tne negative set 

consist of feature vector* ^ comrArted from non-cell 
image patches (see below) and their target classification 
output value 

To obtain 13x15 sized image patches for computing, 
the feature vectors ^* e n<* are an interactive prog***) 
diapleying the digitized microscope fluorescence Images 
and allowing users to select cell centers with the aid ofa 
mouse cursor. Tne set r ncfl is than generated automati- 
cally by randomly selecting Image prints «■ * minimum 
distance of r~ S Pixels from any of the selected calls of 
j-f». p or each of these randomly selected paint* v 
vector jr^* U computed by the Ramc procedure as for T - 
In <ma training *»P *° U-M first one input- output pair 
{x ot y a ) is selected randomly from the tnuning set 
p— (tta,^)!, secondly the best-match node 1? 
froad. and third hs weights are changed according to the 
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PICTURE- z g!gcrKWl!» calculated fttfffl manual mnuHoclick adcrtiCA Of 
120 cran-Mctinncd lymphw=yiCT*- 



fbllowing learning rules 



CD 



II*. 



with c ift , a 0 " 1 ! e A S[0,1] as exponentially decreasing 
learning fltcp &i2**- Looking at the rales, one con ob*avc 
that learning rule (I) is an online version of ft-mean* 
(Moody BfidPaik-eo, 19*9) for portioning $™ « center or 
w>7\ And (2) and (3) adjust a linaar mapping specified by 
vector wg 11 and matrix A* in the vipcinity CVoroni cell) 
around the best match node. 



5, CALCULATION CUT CELL FEATURES 

In the development of a classification syatem a suitable 
feature computation is crucial for the performance of the 
system. In the context of ihia wo* the cell features should 
to 'robust against small changes of ***** intensity and 
curvawne of the fluorescent cetts in tha 1 5 X 15 paiches. 




FIGURE 3 Iflaptim vppmtc) 
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TABLE IT Sl-h,,— 2* T.«11 rtttn^ « ~H. di^l *P ^rf-cc rsccp^ 



Frequency (ft} of T-ocll in rtwuclc 



15.1 

3.1 

2.0 

1.8 

1.3 

1J3 

13 

1. 1 

0.9 

0.9 

0.7 

0.7 

0.7 

0.fi 

0,4 

0.4 

04 

0.2 

0.2 
0-2 
0.2 



0 



CD3 



cr>4 



CDS 



0 
0 
0 

1 

0 
0 
0 

I 
1 
1 
o 

0 
n 
1 
I 

o 

0 

1 

0 
0 
0 
D 
O 
0 



Because the cells are in muscle tissue, their size* intensity 
and shape show ccwi durable variation. One way ro map an 
image point p to a feature vector X S /K R to calculate 
the overlaps of a surrounding image region, of size 
15 x J5vn^asetof6fitew,TOchasGaborfilCers(Dunn 
0f at, 1994; Lee, 1996) or actable doxivsnives of 2- 
dimensional Gaussian; (Rao and Ballard, 1993). The 
disadvantage of such filters is that they contain several 



1 
o 
\ 
i 

0 
D 
D 
0 
0 
1 

0 

I 
1 

0 

1 
1 

0 

1 
] 

n 

o 
a 
i 



o 
0 
0 
0 
0 
0 

L 

1 

1 

0 

0 

\ 

\ 

o 

0 

0 

Q 

1 

0 

0 

0 

0 

1 

1 



CD I I* 

0 
0 
0 
0 

r 

D 
□ 
0 
0 
0 
0 
0 
0 
0 

1 

0 
0 
0 
1 

1 

o 
0 
o 



GDI* 



cms 



o 

0 
0 
0 
(t 

e 
o 
o 
ft 
o 
i 

0 
0 
o 
0 
0 
0 

n 
o 
ii 
o 
i 
o 
o 



o 
o 
o 

0 
0 
0 
0 

o 
i 
i 
o 

0 

] 

o 

0 
0 
1 
tl 
I 

1 
] 

0 
1 
1 



parameters GwKua, orientation*, etc.)* which have io be 
fitted according to the particular application. This is 
difficult rod/or time- consuming for a non-oKpert use*. To 
avoid such problem* wo use here a Principal Component 
Analysts (PCA) on a set of 15 X i5-*ized image patches 
of centered cells, which is a daw-drivco approach* 

to this application the PC A- technique m&ba 6 eigen- 
vectors a* e JR NX (&o called "cigeneellfl") of the 
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covariance roatri* of 15 x 15 image patches. The patches 
are the 15 x 15-sised image regions around the 
handaclected image points of fluorescent cells which 
were selected for training (see above)- The usage of such 
dgenveetois is a well-known technique fcr derecutm tasks 
mtrK andPentland. 1991 ; Varehim «f 1997). A listing 
of the eigenvalues in descending ^V?* 
majority of me variance In the image data is desenbed by 
the six eigeneells of the six highest eigenvalues. Figure 2 
shows 6 eigcncells computed from the (inning Tb^se 
axe taken to s-ncrat* the fi-dmicpslonal feanirc vectors 
for an imago point by scalar multiplication of its 1 5 >c 15 
neighborhood with the six etgencells, 

6\. CELL DETECTION IN FLUORESCENCE 
MICROGRAPHS WITH TF= IAM 

For the fun automatic detection of Ct>26 positive cells and 
other T-cell types in one given micrograph every image 
point p is mapped to its evi dence value p J 1 by calculating 
the LLM output Cfc) S [0< 11 <br its ffaamro vector % of Its 
surrounding 15 X 15 -Image pastfL Mapping evidences of 
all points to their correspondms image position* wo 
achieve the so-called evidence map. A pohrt p c , m the 
evidence map. which ha* the highest evince valine C(x) 
within d 5-pixel «u«us above a given evidence threshold 
^ = 0-5, is the result position of the center of one 
fluorescent cell, so C(x) obeys 

C(p) ><X5 A CQ>) - aiB nwlCV)l dirf % 0) < 5 

Cme input micrograph (A), its evidence map C&> and the 
detected fluorescent cells (C) are shown m Fig- 3. 



7. RESULTS AND PJSCUSSION 



Jh the present paper we have analyzed irrvasive 
T-lympbocytes in muscle tissue for the «P^«°V^ 
DPP IV (CD26). To analyze the cell surface receptor 
expression partem* of these TXelh we used the muW- 
epfcope imaging approach (Schubert, 15*2), ma new form 
worths at chc level of single clearly identifiable **** 
(details will be published elscWhere). Here a 
limited number of seven monoclonal ^tib^esio 
simultaneously detect seven different T-ceUs^ 
proteins, By this approach we have address the role of 
CD26 as a co-stimulatory molecule in dwT-arfl I afltnmon 
caseado. In order to gamine, whBdicr muacle-mvnsivB 
T-cells m sarcoid myopathy do of do not show the 
expected CD26- associated patterns of receptor 
expression. TM» study extends the data obiarodby 
analysis of lymphocyte associated procejns found to be 
expressed in muscle tissue (Schubert «r 1989, 1993; 
Schubert, 1991 ; Haars ei aL 2000). 

The muscle-lnvaMve T-cells were found to be present as 
lowKtensity infiltrates in th» pcxymysium of the muscle 



tissue. SorprisinBly. me majority of these cells were 
Thrive for CD26, indicating tn«aactivarioti ofl'-ceDs via 
me CD26 dependent paibway does not play a key jote m 
this disease. CD19+ B-lymphocytes were rare and 
did not express CD26. Together, only 5- 4% of the TLceW 
expiessed CD26 (see Table n). Tbe-^e cells showed 
substantial variation according to differentia) 
of the six other cell surface receptors (CD2, CD3, CW, 
CDS, CP1 lb). Amoruj the OD26-po*itive ceil fraction, the 
compression of CDS and CD4 or CDS and CD? was seen 
most frequently, whilst few CD26-po*inve nalla ahow*d 
cc-^p^slon of CD4 or COT alone. An muoual T<*n 
type was seen which shows cM&prcs&inn of CD2. CDi l b 
and CD2,6 whilst CM, CD8 and CD3 were absent in 
these cells. Finally the vast majority of T-cclls. which were 
negative for CD26 <- 9S<&) also showed a highly uiiusual 
phenorype characterized either by expression of CD4 
alone, or CD2 alone, whilst compression of the CD3 
receptor complex was restricted to only a minor fraction of 
CD4-po*itivc and CD8-positive T-lyraphocyte* tfor 
summary see Table IT). . 

Together these data show, that muscle- invasive 
T-Mnnhocytes in sarcoid myopathy show a highly 
unusual phenorype at two levels: First, T-lymphocytes m 
the peripheral Wood, that express CD26, are CD^pflMtiVc 
or CpS-positive T-oclls Which are also characterized by 
co-expresrfon of the CD3 complex, as a rule, and mwn of 
these T^ceHs also show co-expression of CD2, which ia 
implicated m sm alternative pathway of T-«dl activanon 
CDavis ai oIm 1999V In the present study it Is shown that 
within the minor fraction of muscle-invasive T-cells. 
which show the expression of CD16 (5. 6% «f all cells), 
only approximately one half of ihesc cells do ahowihe 
CD2/CD3/CD4/CD26 or the CD2/CD3/CD8/CD26 
pbenotype as expected for a fully 'equipped" T-cetl. 
Tlhe other half of The CD26-posit5ve cells express minus* 
variants ofthi* pattern, characterized by omission of one 
or more of the 033/CD2/CD8 ox CO* receptors. 
.IntercstSngly, in these cells. CD26 may occur together 
with the pt M imesrin CD lib- It is unclear at present, 
ivhemer the latter cells are T-cells or macrophages. 
Together, a fraction of the muscle-tnvwuve T-cells show 
both cert surfaces that arc compatible and others that arc 
not compncftlo a of 0320 ^ a co^timolaMry 
Mgnal of T-cell activauoo. m 

Second, the majority of T-eolls, that have uivaded tha 
muscle tissue in sarcoid myopathy are clenr-cut minus- 
variants of the T-cb11 surface expression patterns normttTy 
found in the blood, because these cells in situ only express 
camef CTM or CD2 (together 64% Of all Invasive T-ce.K), 
whilst CD3, CD«, CD1 lb and CD26 are absent. Another 
T-cefl fractioiu which is mbstamially less frequent is the 
CD2*7CT>4 + (13*) and the CD2*/CD3VCD4- T*rt L type 
(5*>\ Only the latter- one would reflect the "normal" jwH 
Saci ^cprc^on pattern of a cm T-celL TJ. 
demonstrates the predominant presence of T-o-ito, 
which, as a rule, lack receptor patterns at the cell surface, 
would be obligatory for the T-ceD activahon 
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cascade. Hence, it ir very unlikely that T-cella invadmg 
the muscle tiaaue in sarcoid Myopathy undergo T-ccU 
activation via the known battery of co-stimulatory 

receptor 5* 

Alternatively we would like to suggest* that the 
heterogeneous "unusual" combinatorial receptor pheno- 
types of the T-cell surface ore involved in differential 
adhesive functions and migratory mechanisms. This view 
is supported by the fact, that CD26, besides its role as a 
co-strmulfftDiy signal In T-cell activation, also exert* 
adhesive functions by binding to collagen (pang et aL, 
1990) Adhesive functions have nlso been assigned to 
CD2, CDS. CD4 end CD lib <Bero1ay *r aL, 1993; Kgott 
and Power, 1993). We therefore suggest, thai all 
phcnoiypes detected in muscle time ra out present 
study (Tabic II) w implicaied Jn cell surface "decision^ 
processed that either tbc the T~eell ar a certain position 
in the tissue or promote T-cell migration* The T-ceH 
may acquire this function by differentially combining 
receptors in a manner illustrated by ibe data presented in 

Tabic II, t _ ..^ 

The neural classifier used to the present study enables 
co analyze and CD26- T-cell pbenoiypee at a 

lame stale. The disease-associated pheootyple data 
presented here and the NCDS as a hlgh-througbpm 
approach could provide important links for mathematical 
modeling T-eell invasion at an Integrated molecular and 
cellular level. 
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